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CLAIMS 

What is claimed is: 

1 . A video surveillance system comprising: 

a sensing unit capable of being operated in a first mode and second mode; and 
a computer system coupled to the sensing unit, the computer system receiving and 
processing image data from the sensing unit, detecting and tracking targets, and 
determining whether the sensing unit operates in the first mode or in the second mode 
based on the detection and tracking of targets. 

2. The video surveillance system of Claim 1, wherein the sensing unit comprises 
means for improving quality of information obtained about a target. 

w • * 

3. The video surveillance system of Claim 2, wherein the means for improving 
quality of information obtained performs at least one of: panning, tilting, and zooming; 
using a super-resolution algorithm; instructing the sensing unit to digitally zoom; 
employing different lenses or filters to view the target; and using a robotic actuator. 

4. The video surveillance system of Claim 3, wherein the computer system includes 
a response module adapted to, in the second mode, obtain improved quality information 
about the target. 

5. The video surveillance system of Claim 2, wherein the computer system provides 
signals to control the means for improving quality of information obtained about a target. 
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6. The video surveillance system of Claim 1, wherein the sensing unit comprises at 
least one of a camera, an infra-red sensor, a thermal sensor, a radio sensor, and an 
algorithmically enhanced camera group. 

7. The video surveillance system of Claim 1, wherein the computer system 
comprises: 

a vision module to process output of the sensing unit; 
an inference module to process output of the vision module; and 
a response module to perform one or more actions based on the output of the 
inference module. 

8. The video surveillance system of Claim 7, wherein, in the first mode, the vision 
module detects at least one of blobs and targets. 

9. The video surveillance system of Claim 7, wherein, in the first mode, the vision 
module comprises: 

a segmentation module to separate background pixels from foreground pixels; 

a blobizer to receive the foreground pixels from the segmentation module and to 
group connected foreground pixels into coherent blobs; 

a target tracker to process the coherent blobs, determine when they are targets, 
and to obtain position information for each target; 

a classifier to determine a target type for each target; and 
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a primitive generation module to generate summary statistics to be sent to the 
inference module. 

10. The video surveillance system of Claim 7, wherein, in the first mode, the 
inference module determines when at least one specified condition has been either met or 
violated. 

1 1 . The video surveillance system of Claim 7, wherein the response module is 
adapted to perform at least one of the following: sending an e-mail alert; sounding an 
audio alarm; providing a visual alarm; logging an alert to a persistent medium; making a 
telephone call; transmitting a wireless message to a personal digital assistant; enabling a 
mechanical actuator; and providing position information to another sensing unit. , 

"■ - . 

12. The video surveillance system of Claim 7, wherein, in the second mode, the vision 
module employs at least one of target motion, color and edge information derived from 
the image data to determine target location. 

13. The video surveillance system of Claim 7, wherein, in the second mode, the 
vision module comprises: 

a motion detection module that determines camera motion and segments target 
motion after having factored out camera motion; 

a color histogram module that calculates a color histogram for a region near each 
pixel in an area near a target; 
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an edge detection module that determines edges in any given image; 

a template matching module that uses motion segmentation and color information, 
as well as an edge structure of a target template, to determine a best guess of the targets 
position; and 

a primitive generation module that generates summary statistical information 
based on the best guess of the target's position. 

14. The video surveillance system of Claim 13, wherein the motion detection module 
comprises: 

a corner detection module to find interesting points; 

a search module to find matches for interesting points between successive images; 

a confidence value assignment module to assign confidence values to the matches 
of the interesting points; 

a robust averaging module" to determine an estimate from a set of high confidence 
matches; 

a warping module to warp one successive image to another successive image for 
direct comparison of the images; and 

a subtraction module, which subtracts the warped image from a current image to 
determine which pixels have moved. 

15. The video surveillance system of Claim 7, wherein, in the second mode, the 
inference module determines whether to switch back to the first mode by monitoring at 
least one of how long the system has been in the second mode, the position of the target 
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in the image, the position of the target in a world coordinate frame, and one or more 
external stimuli. 

16. The video surveillance system of Claim 7, wherein, in the second mode, the 
response module receives higher quality information about the target than in the first 
mode. 

17. The video surveillance system according to Claim 1, further comprising: 

a second sensing unit capable of being operated in the first mode and second 
mode; and 

a second computer system coupled to the second sensing unit and to the computer 
system, the second computer system receiving and processing image data from the 
second sensing unit, detecting and tracking targets, and determining whether the second 
sensing unit operates in the first mode or in the second mode based on the detection and 
tracking of targets, 

wherein the computer system and the second computer system cooperate to 
coordinate handoff between the sensing unit and the second sensing unit to track at least 
one target. 

18. A method of operating a video surveillance system, the video surveillance system 
including at least one sensing unit capable of being operated in first and second modes, 
the method comprising the steps of: 

operating a sensing unit in the first mode to scan for targets; 
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processing image data from the sensing unit in a first mode to detect the presence 
of an interesting target; 

upon detecting an interesting target, operating the sensing unit in the second mode 
to track the interesting target and to improve the quality of information about the 
interesting target over the information that can be obtained in the first mode; and 

processing image data from the sensing unit in a second mode to track the target 
by sending at least one of pan, tilt, and zoom commands to the sensing unit. 

19. The method of Claim 1 8, wherein the step of operating the sensing unit in the first 
mode comprises the step of performing at least one of: 

operating the sensing unit in a wide-angle mode to scan for targets. 

20. The method of Claim 1 8, wherein the step of operating the sensing unit in the 
second mode comprises the steps of: 

obtaining higher quality imagery of a target than that obtained by operating the 
sensing unit in the first mode through the use of at least one of the following: 
panning, tilting, and zooming; 
using super-resolution algorithms; 
digitally zooming into the image; 
employing different lenses or filters; and 
using a robotic actuator. 
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2 1 . The method of Claim 1 8, wherein the step of processing image data from the 
sensing unit in a first mode comprises the steps of: 

processing the image data with a vision module to detect and locate at least one 
target; and 

determining if at least one predetermined condition has been violated by at least 
one target. 

22. The method of Claim 21, wherein the step of processing the image data with a 
vision module comprises the step of: 

geo-locating the at least one object in 3D space. 

23. The method of Claim 21, wherein the step of processing the image data with a 
vision module comprises the steps of: c 

classifying pixels in the image information as background pixels or foreground 
pixels; and 

using the foreground pixels to determine at least one blob. 

24. The method of Claim 23, further comprising the step of tracking at least one 
possible target based on at least one blob. 

25. The method of Claim 24, wherein the step of tracking comprises the steps of: 
determining when at least one blob merges or splits into one or more possible 

targets; and 
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performing at least one of: 

filtering and predicting the image location of at least one of the possible targets; 

and 

calculating a 3D position of at least one of the possible targets if calibration 
information is available. 

26. The method of Claim 24, further comprising the step of classifying at least one 
possible target by target type into categories. 

27. The method of Claim 24, further comprising the step of providing summary 
statistics to aid in the step of determining if at least one predetermined condition has been 
violated by at least one object. 

28. The method of Claim 27, wherein the summary statistics include at least one of 
the following: an estimate of the targets position in image space; the target's position in a 
world coordinate frame; the target's size in image space; the target's size in the world 
coordinate frame; the target's velocity in image space; the target's velocity in the world 
coordinate frame; the target's trajectory; the target's color; the color of at least one 
subcomponent of the target; a bounding box for the target in an image; and at least one 
estimate of location and size of at least one body part, in the case where the target is 
human. 
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29. The method of Claim 1 8, wherein the step of processing image data from the 
sensing unit in a second mode, if the second mode involves camera motion, comprises the 
steps of: 

segmenting the target from the background by detecting motion of the sensing 
unit and factoring it out; 

calculating color histogram information in a region of at least some pixels of 
image data from the sensing unit; 

calculating an edge image; and 

using template matching, to integrate the results of the steps of motion 
segmentation, edge detection and color histogram formation, to obtain an estimate of the 
target's position. 

30. The method of Claim 29, wherein the step of processing image data from the 
sensing unit in a second mode further comprises the step of: 

generating summary statistics based on various features of the target. 

31 . The method of Claim 30, wherein the summary statistics include at least one of 
the following: an estimate of the target's position in image space; the target's position in a 
world coordinate frame; the target's size in image space; the target's size in the world 
coordinate frame; the target's velocity in image space; the target's velocity in the world 
coordinate frame; the target's trajectory; the target's color; the color of at least one 
subcomponent of the target; a bounding box for the target in an image; and at least one 
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estimate of location and size of at least one body part, in the case where the target is 
human. 

32. The method of Claim 29, wherein the step of processing image data from the 
sensing unit in a second mode further comprises the steps of: 

monitoring a length of time in which the system has been in the second mode; 

monitoring the various summary statistics to detect if any target characteristic has 
met a predetermined rule for switching to the first mode; 

monitoring for any external stimuli indicating that the system should switch to the 
first mode; and 

if any predetermined conditions are met in any of the monitoring processes,, 
switching back to the first mode. - 

33. The method of Claim 18, further comprising the steps of: 

determining a best shot of the target while operating the visual module in either of 
the first and second modes; and 

delivering the best shot back to the user through the response module. 

34. The method of Claim 1 8, further comprising the steps of: 

if the target exits the field of view of the sensing unit, directing a second sensing 
unit to scan for and track the target. 
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35. The method of Claim 34, wherein the step of directing a second sensing unit to 
scan for and track the target comprises the steps of: 

receiving, by the second sensing unit, information about a last known position of 
the target from the sensing unit; 

operating the second sensing unit in the first mode to scan for the target, focusing 
on a region near the last known position; 

processing image data from the second sensing unit in the first mode to detect the 
presence of a target; 

upon detecting a target, operating the second sensing unit in the second mode to 
track the target; and 

processing image data from the second sensing unit in the second mode to track 
the target by sending at least one of pan;: tilt, and zoom commands to the second sensing 
unit, 

wherein, if the target is not detected within a predetermined time period in the 
step of operating the second sensing unit in the first mode to scan for the target, focusing 
on a region near the last known position, the second sensing unit switches to scanning for 
targets in its entire field of coverage. 

36. A computer-readable medium containing software implementing the method of 
Claim 18. 

37. A video surveillance system, comprising: 
at least one sensing unit; 
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at least one computer system; and 

the computer-readable medium of Claim 36. 

38. The method of Claim 18, wherein, if operating the sensing unit in the second 
mode involves camera motion, processing image data from the sensing unit in a second 
mode comprises: 

(a) performing corner detection to find interesting points; 

(b) searching for matches for interesting points in successive images; 

(c) assigning confidence values to the matches of the interesting points; 

(d) employing a robust averaging method to determine an estimate from a subset 
of the matches determined to have high confidence values; 

(e) warping a second successive image towards a first successive image; and 

(f) subtracting the warped image from a current image to determine which pixels 
have moved. 

39. The method of Claim 38, wherein processing image data from the sensing unit in 
a second mode further comprises: 

performing steps (a)-(f) on reduced resolution images; and 
performing steps (a)-(f) on full resolution images, utilizing the results of 
performing steps (a)-(f) on reduced resolution images to reduce the computational burden 
by limiting the searching to only a subset of the full resolution images. 
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40. The method of Claim 38, wherein processing image data from the sensing unit in 
a second mode further comprises: 

using three-frame differencing to alleviate false motion. 

41 . A motion detection module to determine camera motion, comprising: 
a corner detection module to find interesting points; 

a search module to find matches for interesting points between successive images; 

a confidence value assignment module to assign confidence values to the matches 
of the interesting points; 

a robust averaging module to determine an estimate from a set of high confidence 
matches; 

a warping module to warp one successive image to another successive image for 
direct comparison of the images; and 

a subtraction module, which subtracts the warped image from a current image to ■* 
determine which pixels have moved. 

42. The motion detection module of Claim 41, wherein each module operates first on 
reduced resolution image data and then on full resolution image data, using the results of 
operating on reduced resolution image data to reduce a computational burden. 

43. The motion detection module of Claim 41, wherein the subtraction module uses 
three-frame differencing to alleviate false motions. 
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44. A video-based security system, comprising the video surveillance system 
according to Claim 1. 

45. A video-based security system, comprising the video surveillance system 
according to Claim 37. 

46. A method of operating a video-based security system, comprising the method 
according to Claim 18. 

47. An experiment monitoring system, comprising the video surveillance system 
according to Claim 1. 

48. An experiment monitoring system, comprising the video surveillance system 
according to Claim 37. 

49. A method of monitoring an experiment, comprising the method according to 
Claim 18. 

50. The method according to Claim 49, further comprising: 

focusing in on at least one predetermined behavior of at least one subject of the 
experiment. 
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51. A sporting event monitoring system, comprising the video surveillance system 
according to Claim 1. 

52. A sporting event monitoring system, comprising the video surveillance system 
according to Claim 37. 

53. A method of monitoring a sporting event, comprising the method according to 
Claim 18. 

54. The method according to Claim 53, further comprising: 

detecting at least one predetermined behavior of a participant in the sporting 

event. 

55. A marketing information gathering system, comprising the video surveillance 
system according to Claim 1 . 

56. A marketing information gathering system, comprising the video surveillance 
system according to Claim 37. 

57. A method of gathering marketing information, comprising the method according 
to Claim 18. 

58. The method according to Claim 57, further comprising: 
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monitoring at least one behavior of at least one subject. 

The method according to Claim 58, wherein said monitoring comprises: 
detecting interest in a given product. 

The method according to Claim 59, wherein said detecting comprises: 
detecting when a customer reaches for said given product. 
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